# Gesture Recognition Algorithm

***Copyright © Quectel Wireless Solutions Co., Ltd. 2026. All rights reserved.***

---

This guide walks through building a desktop GUI gesture-recognition project from scratch on Quectel Pi M2 Debian, using the gesture-recognition algorithm SDK.

The interface uses a three-column layout:

- **Left column**: Algorithm modules (currently “Gesture Recognition”)

- **Center column**: Title, live display, and controls

- **Right column**: Device status (RAM / ROM / CPU)

Two input modes are available:

- **USB camera**: Open a USB camera and recognize gestures in its live video stream.

- **Local image**: Select a local image and recognize gestures offline.

## Algorithm Capabilities

| **Item** | **Description** |
| --- | --- |
| Algorithm type | Cascaded hand detection and gesture classification |
| Detection model | board6_RK3576_Ubuntu_hand_det.engine (640² input, hand bounding boxes) |
| Classification model | board6_RK3576_Ubuntu_gesture_cls.engine (224² input, 13 classes) |
| Input | Contiguous BGR memory (cv::Mat.data, rows × columns × 3) |
| Output | One DetRes per hand (xywh box, class label, and confidence) |

**Example gesture labels**: `like`, `dislike`, `call`, `fist`, `four`, `mute`, `ok`, `one`, `palm`, `three`, `yeah`, and `other`.

## Prerequisites

### Obtain the Resources

| **Name** | **Description** |
| --- | --- |
| gesture_RK3576 (algorithm SDK package) | Gesture-recognition package containing the public header, shared libraries, third-party runtimes, models, and sample images |
| [gesture_project](<https://github.com/Quectel-Pi/demo-edge-algorithms>) (demo project) | Example project from this guide, including source code, Makefile, and three scripts for copying dependencies, building, and running |

Key directories in the `gesture_RK3576` SDK package:

```bash
gesture_RK3576/
├── include/gesture.h            # Public algorithm header (the only header required for integration)
├── lib/libgesture.so            # Main algorithm library (cascaded detection and classification)
├── 3rd_party/
│   ├── modelinfer/              # Inference engine libinference.so + authorization libauthorization.so
│   ├── rknn/librknnrt.so        # RKNN hardware runtime
│   └── opencv/                  # Bundled OpenCV world library (headers + libopencv_world.so)
├── models/                      # Detection and classification .engine models
├── images/hand_input.jpg        # Sample image for acceptance testing
└── example.cpp                  # Official minimal usage example
```

### Hardware and System Requirements

| **Item** | **Requirement** |
| --- | --- |
| Main board | Quectel Pi M2 |
| System | Debian / Ubuntu aarch64 with a desktop environment (Wayland or X11) |
| Peripheral | USB camera (required for live recognition) |
| Connection | The device can be accessed from a PC using adb shell |

### Software Requirements

The demo is built natively on Quectel Pi M2. Make sure the following software is installed:

```bash
g++ --version          # C++17 compiler
make --version
pkg-config --exists Qt5Widgets Qt5Gui Qt5Core && echo "Qt5 OK"   # Qt5 (LGPL) for display
v4l2-ctl --version     # Enumerate and troubleshoot USB camera nodes
```

> **UI implementation**: The display layer uses the system-provided **Qt5**, so it displays directly on the desktop and supports full-screen mode. Capture, inference, and drawing use the project-bundled OpenCV world library, isolated from the system OpenCV to avoid version conflicts.

## Project Structure

```bash
gesture_project/
├── src/
│   └── gesture_cam_demo.cpp     # Main demo (Qt UI + inference worker thread)
├── include/                     # Populated by setup: SDK public header gesture.h
├── lib/                         # Populated by setup: libgesture.so
├── third_party/                 # Populated by setup: modelinfer / rknn / opencv runtimes
├── models/                      # Populated by setup: detection and classification .engine models
├── images/                      # Populated by setup: sample image hand_input.jpg
├── Makefile                     # Build rules
├── build.sh                     # One-command build
├── setup_from_sdk.sh            # Copy dependencies from the SDK package into this project
├── run.sh                       # Configure environment variables and launch
└── README.md                    # Documentation
```

`include/`, `lib/`, `third_party/`, `models/`, and `images/` are generated by `setup_from_sdk.sh` from the original SDK package.

## Core Code Walkthrough

The demo uses the `Gesture` class exposed by the SDK's public `gesture.h` header. Its lifecycle is: **construct →`set_param`****(set thresholds) →`init`****(load models and authorize) →`inference`****(infer each frame) →`uinit`****(release resources)**.

The source file is `src/gesture_cam_demo.cpp`. The following sections explain key snippets in call order.

### Initialization

```cpp
Gesture g;
float conf = 0.25f, iou = 0.45f, det_thr = 0.0f, cls_thr = 0.20f;
g.set_param(conf, iou, det_thr, cls_thr);          // Four reference parameters: detection confidence / NMS / detection filter / classification threshold

char det_model[] = "models/board6_RK3576_Ubuntu_hand_det.engine";
char cls_model[] = "models/board6_RK3576_Ubuntu_gesture_cls.engine";
char license_buf[512] = "authorization.lic";
if (const char* license_path = std::getenv("GESTURE_LICENSE_FILE")) {
    if (license_path[0]) std::snprintf(license_buf, sizeof(license_buf), "%s", license_path);
}
// p_auth is the algorithm authorization code, supplied through GESTURE_AUTH_CODE or the command line (not bundled with the SDK).
char auth_buf[512] = {0};
const char* auth = std::getenv("GESTURE_AUTH_CODE");
char* p_auth = nullptr;
if (auth && auth[0]) { std::snprintf(auth_buf, sizeof(auth_buf), "%s", auth); p_auth = auth_buf; }
GESTURE_RES_CODE rc = g.init(det_model, cls_model, license_buf, 3, p_auth);
if (rc != GESTURE_PROCESS_OK) { /* Initialization failed: check the authorization code and model paths. */ }
```

Parameter descriptions:

- **`set_param(conf, iou, det_thr, cls_thr)`**: All four parameters are passed by reference.  
    - `conf`: Hand **detection** confidence threshold; default `0.25`. Candidates scoring below this value are discarded during detection. Increase it to reduce false positives or decrease it to detect more hands.  
    - `iou`: IoU threshold for detection-box **NMS (non-maximum suppression)**; default `0.45`. When two boxes overlap beyond this threshold, only the higher-scoring box is retained, preventing duplicate boxes for one hand.  
    - `det_thr`: **Output-filter** threshold for detection results; default `0.0` (no filtering). Applies a second lower-bound check to the final `det_score`.  
    - `cls_thr`: **Classification** threshold; default `0.20`. If the top-1 classification score is below this value, `label` is set to `other` (uncertain gesture).

- **`init(p_det_model, p_cls_model, p_license_file, module, p_auth_code)`** loads both models and performs authorization. The five arguments are:  
    - `p_det_model`: Path to the hand **detection** model (`hand_det.engine`, 640² input).  
    - `p_cls_model`: Path to the gesture **classification** model (`gesture_cls.engine`, 224² input, 13 classes).  
    - `p_license_file`: Writable path for the algorithm license file. The project's `run.sh` supplies an absolute path inside the project through `GESTURE_LICENSE_FILE`.  
    - `module`: Platform identifier; pass `3` for RK3576.  
    - `p_auth_code`: Algorithm authorization code (not bundled with the SDK); supply it through `GESTURE_AUTH_CODE` or an argument.

- **Return code`GESTURE_RES_CODE`**: `0` = `GESTURE_PROCESS_OK` (success); `-1` = initialization failed (authorization/model path); `-2` = not initialized; `-3` = inference failed; `-4` = invalid image/parameters. Check for `GESTURE_PROCESS_OK` before running inference.

- **`get_status(opt)`** queries status: `get_status(0)` returns initialization status (`0` means success); `get_status(1)` returns authorization status (`0` means fully authorized).

### Inference

Call `inference` once per frame (from either the camera or a local image). The SDK performs the full detection → crop → classification cascade and returns one result per hand.

```cpp
std::vector<DetRes> results;
// Arguments: BGR data pointer, rows (height), columns (width), channels (must be 3), output result vector
GESTURE_RES_CODE rc = g.inference(frame.data, frame.rows, frame.cols, frame.channels(), results);
if (rc != GESTURE_PROCESS_OK) { /* Inference failed. */ }

for (const auto& r : results) {
    // r.x, r.y, r.width, r.height : Detection box (xywh in original-image pixels)
    // r.det_score                 : Detection confidence
    // r.class_id                  : Classification ID (0–12)
    // r.label                     : Gesture label (for example, "fist")
    // r.cls_score                 : Classification confidence
    cv::rectangle(frame, cv::Rect(r.x, r.y, r.width, r.height), cv::Scalar(0,255,0), 2);
}
```

**Input requirements**:

- Input must be **contiguous BGR memory** (`cv::Mat` is contiguous by default; if `isContinuous()` is false, call `clone()` before passing `.data`).

- `channels` must be `3`. Convert grayscale images to BGR with `cv::cvtColor` first.

- The SDK automatically handles BGR → RGB, letterbox resizing to 640², `/255` normalization, and int8 quantization. **No preprocessing is required in the caller.**

**Cascade performed inside the SDK**:

1. Run the full image through `hand_det` to detect all hands, then apply NMS to the candidates.

2. Crop each hand with 10% surrounding context, resize it to 224², and normalize with ImageNet mean and standard deviation.

3. Run `gesture_cls` to get 13-class softmax scores and select top-1. If its score is below `cls_thr`, label the result `other`.

**Output structure`DetRes`** (one per hand):

| **Field** | **Meaning** |
| --- | --- |
| x / y / width / height | Detection box in original-image pixels (top-left corner + width and height) |
| det_score | Detection confidence (confidence in the hand box) |
| class_id | Gesture class ID (0–12) |
| label | Gesture name (for example, fist, ok, or palm) |
| cls_score | Classification confidence (top-1 softmax score) |

> Coordinates are mapped back to the original image and can be drawn directly with `cv::rectangle` / `cv::putText`. An empty `results` vector means no hand was detected in the current frame.

### Release

```cpp
g.uinit();   // Release internal engine resources
```

## Deployment and Run

### Transfer the Files

The PC-side examples use Windows PowerShell. First push the directories to `/tmp` on the device with `adb push`:

```bash
# Push the directories to /tmp on Quectel Pi M2 first.
adb push gesture_RK3576 /tmp/
adb push gesture_project /tmp/
```

After the push completes, open a terminal in the Quectel Pi M2 desktop session and run the following as a regular user.

```bash
# $HOME and $$USER are expanded by the device shell. Do not run this in a root adb shell,
# where they would refer to the root account.
sudo mv /tmp/gesture_RK3576 /tmp/gesture_project "$HOME"/
sudo chown -R "$$USER:$$USER" "$$HOME/gesture_RK3576" "$$HOME/gesture_project"
cd "$HOME/gesture_project" && chmod +x *.sh
```

### Install the Build Environment and Tools

Run on the Quectel Pi M2:

```bash
# Install make, g++, and other development tools.
sudo apt update && sudo apt install -y build-essential pkg-config qtbase5-dev
```

### Copy Dependencies

`setup_from_sdk.sh` copies headers, shared libraries, third-party runtimes, models, and sample images from the SDK package into the corresponding project directories:

```bash
cd "$HOME/gesture_project"
sh setup_from_sdk.sh "$HOME/gesture_RK3576"   # Omit the argument to use ~/gesture_RK3576 by default.
```

### Build

```bash
sh build.sh
# Output: $HOME/gesture_project/gesture_cam_demo
```

> `build.sh` invokes `make`: it uses the system Qt5 for the UI, then links the project-local `libgesture.so` and bundled OpenCV world library to produce the executable.

### Run

Run from a **desktop terminal** on Quectel Pi M2:

```bash
./run.sh                # Automatically detect a USB camera
./run.sh /dev/video73   # Specify a USB camera node (skips automatic detection)
```

> We recommend launching from a terminal in the device's desktop session. If launching through `adb shell`, switch to the desktop login user and make sure `XDG_RUNTIME_DIR`, `WAYLAND_DISPLAY`, or `DISPLAY`/`XAUTHORITY` matches the desktop session.

### Use the UI

The window has three columns: algorithm modules on the left, display and controls in the center, and device status on the right.

| **Control** | **Description** |
| --- | --- |
| **[Open Camera]** | Open/close the USB camera and recognize gestures in real time; closing it clears the display. |
| **[Local Image]** | Open a file picker and run offline recognition on a selected image. |
| **[Full Screen]** | Toggle between full-screen and windowed mode. |
| **[Quit]** | Close the program. |
| Keyboard shortcuts | Esc/Q to quit; F for full-screen mode. |

The **device status panel** on the right refreshes once per second:

| **Metric** | **Description** |
| --- | --- |
| RAM | Used percentage and used/total memory (MB) |
| ROM | Root filesystem usage percentage and used/total capacity (GB) |
| CPU | CPU utilization percentage |

## Authorization

The SDK uses two authorization layers, engine first and algorithm second. The integrator only needs to provide the **algorithm authorization code** (`p_auth_code`); the other components are fixed in the package:

- Supply the code through `GESTURE_AUTH_CODE` (already exported by `run.sh`; replace its value with your own code).

- `GESTURE_LICENSE_FILE` specifies the license-file path. By default, `run.sh` sets it to the absolute path `authorization.lic` inside the project. The engine license file `ql_ans_license.txt` is also stored in the project. These files are generated after the first successful online authorization and reused on later launches.

- A valid algorithm code grants full authorization (`get_status(1)=0`).

- If authorization returns `100010` and the SDK confirms that the authorization server is unreachable, the algorithm enters a **30-minute trial** (`get_status(1)=100010`; expiration maps to `100002`).

- An empty code or an incorrect code while online causes `init` to fail; neither case enters the algorithm trial.

Status queries: `get_status(0)` returns algorithm initialization status (`0` means success); `get_status(1)` returns authorization status.

## Demo Video

<div class="video-preview">
  <video width="100%" controls preload="none">
    <source src="videos/video_Xf4Nb166jokpwJxHmPWcJym2nye.mp4" type="video/mp4">
    Your browser does not support the video tag.
  </video>
  <div class="video-caption">
    <strong>gesture-demo</strong>
  </div>
</div>


## Troubleshooting

| **Issue** | **Possible cause** | **Solution** |
| --- | --- | --- |
| Initialization fails with rc=-1 | Authorization code is empty/incorrect, or a model path is wrong | Set GESTURE_AUTH_CODE; verify that both .engine files exist under models/. |
| Window does not appear | Not running in a desktop session / QT_QPA_PLATFORM does not match | Run in the device's desktop session, or let run.sh detect Wayland/xcb automatically. |
| Camera cannot be opened | USB camera is disconnected / specified node is wrong | Automatic mode probes for USB capture nodes. Use v4l2-ctl --list-devices to check, then run ./run.sh /dev/videoN. |
| Link or runtime error: .so not found | LD_LIBRARY_PATH is not set | Launch with run.sh (which configures it), or export the project lib and third_party/* paths manually. |
| No recognition results | No gesture in view / confidence is too low | Keep the hand clearly in frame; adjust the detection confidence and classification threshold in set_param if needed. |